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Abstract. Static analysers search for overapproximating proofs of safety 
commonly known as safety invariants. Fundamentally, such analysers 
summarise traces into sets of states, thus trading the ability to distin¬ 
guish traces for computational tractability. Conversely, static bug finders 
(e.g. Bounded Model Checking) give evidence for the failure of an asser¬ 
tion in the form of a counterexample, which can be inspected by the 
user. However, static bug finders fail to scale when analysing programs 
with bugs that require many iterations of a loop as the computational 
effort grows exponentially with the depth of the bug. We propose a novel 
approach for finding bugs, which delivers the performance of abstract in¬ 
terpretation together with the concrete precision of BMC. To do this, 
we introduce the concept of danger invariants - the dual to safety in¬ 
variants. Danger invariants summarise sets of traces that are guaranteed 
to reach an error state. This summarisation allows us to find deep bugs 
without false alarms and without explicitly unwinding loops. We present 
a second-order formulation of danger invariants and use the solver de¬ 
scribed in [1] to compute danger invariants for intricate programs taken 
from the literature. 


Keywords: static bug finding, deep bugs, second-order logic, trace summarisation, 
program synthesis. 


1 Introduction 

Safety analysers search for proofs of safety commonly known as safety invariants 
by overapproximating the set of program states reached during all program exe¬ 
cutions. Fundamentally, they summarise traces into abstract states, thus trading 
the ability to distinguish traces for computational tractability [2]. Consequently, 
safety analysers may generate bug reports that do not correspond to actual er¬ 
rors in the code (i.e. false alarms). This is illustrated in Figure 1. False alarms 
are the primary barrier to the adoption of static analysis technology outside 
academia. Triage of true errors and false alarms is a tedious and difficult task, 
and there are reports that developers fare no better than coin tossing [3]. 

Conversely, static bug finders such as Bounded Model Checking (BMC) 
search for proofs that safety can be violated. Dually to safety proofs, we will call 
these danger proofs. Static bug finders have the attractive property that once 
an assertion fails, a counterexample trace is returned, which can be inspected 
by the user [4]. The counterexample is thus the proof that an assertion viola¬ 
tion occurs. In order to construct such a danger proof, bounded model checkers 
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Fig. 1: Generation of false Fig. 2: Iterative unwinding of the 

alarms transition relation 


compute underapproximations of the reachable program states by progressively 
unwinding the transition relation. The downside of this approach is that static 
bug finders fail to scale when analysing programs with bugs that require many 
iterations of a loop. For illustration, Figure 2 depicts the successive unwinding 
of the transition relation, where progressively larger sets of reachable program 
states are labelled with letters from A to F. The computational effort required 
to discover an assertion violation (i.e. to obtain an intersection with the small 
ellipse labelled “error states”) typically grows exponentially with the depth of 
the bug. 

Notably, the scalability problem is not limited to procedures that imple¬ 
ment BMC. Approaches based on a combination of over- and underapproxima¬ 
tions such as predicate abstraction [5] and lazy abstraction with interpolants 
(LAwI) [6] are not optimised for finding deep bugs either. The reason for this is 
that they can only detect counterexamples with deep loops after the repeated 
refutation of increasingly longer spurious counterexamples. The analyser first 
considers a potential error trace with one loop iteration, only to discover that 
this trace is infeasible. Consequently, the analyser increases the search depth, 
usually by considering one further loop iteration. This repeated unwinding suf¬ 
fers from the same exponential blow-up as BMC. 

Danger proofs. In this paper we propose a novel representation of a danger proof 
based on trace summarisation. We propose to merge the two core concepts of 
safety analysers based on abstract interpretation and bug finders: trace summari¬ 
sation (for scalability purposes) and counterexample generation (for precision). 
The intuition is that summarising traces is permissible as long as the summary 
is guaranteed to contain at least one feasible counterexample trace (i.e. a trace 
that starts in an initial state and reaches an error state). 

The resulting summary is a dual of a safety invariant, which we refer to as 
a danger invariant. As opposed to safety invariants, danger invariants do not 
necessarily include all the reachable program states, but must contain at least 
one feasible execution trace. A danger invariant may encompass multiple paths 
through the program, but contains enough information to directly read off a 
concrete error trace. The danger invariant therefore amounts to a concise proof 
that such an error trace exists. 
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From a practical point of view, danger invariants will allow the development 
of bug finding techniques that do not require explicit loop unwinding. We empiri¬ 
cally show that danger invariants improve the scalability of bug finding, enabling 
the detection of deep bugs. From a theoretical point of view, we propose a dual 
to over-approximating safety invariants, which are at the core of safety analysis. 
Over-approximating invariants have received an enormous quantity of research 
ever since the seminal work of Cousot and Cousot [2]. The equivalent of this 
body of work for invariants biased towards bug finding is missing. 

While the main technique that we use in this paper to compute danger in¬ 
variants is based on program synthesis (Section 3.1), given that the trace sum¬ 
marisation concept is common to both safety and danger invariants, we also 
investigate how methods used for safety invariant generation can be adapted for 
danger invariants (Section 5). 

Contributions: 

— To the best of our knowledge, we propose the hrst formulation of a 
concise proof of the existence of an error trace that allows trace sum¬ 
marisation without false alarms. 

— We show that danger invariants improve the scalability of bug hnding 
by using the second-order solver in [1] to infer danger invariants for 
programs with deep bugs. 

— We investigate how techniques used for inferring safety invariants can 
be adapted for danger invariants. 

— We generate danger invariants for a set of benchmarks taken from the 
literature. 


2 From Counterexamples to Danger Invariants 

We represent a program P as a transition system with state space X and tran¬ 
sition relation T C X x X. For a state x G X with T{x,x'), x' is said to be a 
successor of x under T. We denote initial states by I and error states by E. 

Definition 1 (Execution Trace). A program trace {xq ... Xn) is a (potentially 
infinite, in which case n = oj) sequence of states, such that any two successive 
states are related by the program’s transition relation T, i.e. 

VO < i < n.T{xi,Xi+i) . 

Definition 2 (Counterexample). A finite execution trace (xq ... x„) is a coun¬ 
terexample iff Xq is an initial state, xq G I, and x„ is an error state, x„ G E. 

A counterexample is an instance of a danger proof: it provides evidence that 
an error state will be reached in some program execution. The question we try to 
answer in this paper is whether we can derive a more compact representation of a 
danger proof that does not require us to explicitly write down every intermediate 
state. Similarly to safety invariants, we obtain such a compact representation by 
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summarising traces. While this approach may involve overapproximation, the 
tricky part is retaining enough precision to ensure that a counterexample trace 
exists. In the rest of the section, we will explain our formulation by starting 
from a safety invariant and progressively adjusting it to show the existence of a 
counterexample. For this purpose, we will refer to loops of the form L(G, T, I, A) 
shown in Figure 3, which we encode with predicates for the initial states: I{x), 
guard: G{x), body: T{x,x') and assertion: A{x). 


assume (/); 
while (G) T; 
assert (^); 

Fig. 3: The general form of a loop 


Safety Invariants. Intuitively, a safety invariant is a set of states S that includes 
every state reachable via zero or more iterations of the loop, and which excludes 
all error states. More formally: 

Definition 3 (Safety Invariant). A predicate S is a safety invariant for the 
loop L{I, G, T, A) iff it satisfies the following criteria: 

Vx.I{x) —>■ S{x) (I) 

Vx, x'.S{x) A G{x) A T{x, x') —>■ S{x') (2) 

Vx.S{x) A “'G(x) —> A{x) (3) 


2.1 Prom Safety to Danger 

It is well known that Definition 3 captures the notion of a safety invariant, and 
that if a predicate S exists that satisfies these three criteria, then the loop L 
is safe. We will now consider what the dual notion of a danger invariant might 
look like, and identify the criteria defining it. 

Let us begin by considering what happens if we take the natural step of 
replacing criterion 3 with its complement: if we exit the loop in an S'-state, we 
would like the assertion to fail. This gives us the following definition: 

Definition 4 (Doomed Loop Head). The head of the loop L{I,G,T,A) is a 
doomed point [7] iff there exists a predicate S' satisfying: 

Vx./(x) —>■ S'{x) 

Vx, x'.S'{x) A G(x) A T(x, x') —>■ S'{x') 

\/x.S'{x) A “'G(x) ^ ^A{x) 


( 4 ) 

( 5 ) 

( 6 ) 
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The term “doomed program point” was introduced in [7] and denotes a pro¬ 
gram location that, whenever reached, will inevitably lead to an error regardless 
of the state in which it is reached. Definition 4 applies this notion to the head 
of the loop L: the predicate S' provides proof that the head of the loop L is a 
doomed point, so if such an S' exists then the loop will certainly fail. However, 
this definition is overly restrictive: in practice, for most unsafe programs such 
an S' does not exist. 

For illustration, the program in Figure 4a is unsafe (any execution starting 
with X > 10 will lead to the assertion failing) but since there are some initial 
states that do not lead to an error (i.e. every state with x < 10) there are no 
doomed points. Thus, we weaken Definition 4 by introducing doomed program 
states: we weaken criterion 1 to say that there must be some initial state leading 
to an error rather than every initial state. This gives us the following: 


Definition 5 (Doomed Program State). There is traee containing an error 
state, starting from the head of the loop L{I,G,T,A) if a predicate S" exists 
satisfying: 

3xo./(xo) ^ S"ixo) (7) 

Vx, x'.S"{x) A G'(x) A T(x, x') ^ S"{x') (8) 

\/x.S"{x) A “'G(x) —> -iH(x) (9) 


X = * ; 

while (x < 10) { 

X++; 

} 

assert (x 10); 


(a) An unsafe loop with 
no doomed program 
points. 


X = 0; 

while (x < 10) { 

if(*) break; 
X-I-+; 

} 

assert (x 10); 


(b) An unsafe loop with 
no doomed states. 


x = 0; 
y = 0; 

while (x < 10) { 

y+-l-; 

} 

assert(x < 10); 


(c) A safe program with 
no finite error traces. 


Fig. 4: Illustrative programs - * means nondeterministic choice. 


Definition 5 weakens the notion of a doomed location. Rather than requiring 
every trace including the doomed location to reach an error, we only require that 
there is some doomed initial state, i.e. every trace including that state will reach 
an error. However, this is still too strong a condition! Figure 4b shows an unsafe 
program which has no doomed states - every state in the loop has some successor 
that leads to a state in which the assertion holds (i.e. every trace in which the 
break is never executed). To work around this, we can weaken criterion 2 to say 
that each state must have some successor leading to an error. 
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Definition 6 (Partial Danger). There is a trace containing an error state, 
starting from the head of the loop L{I, G, T, A) if a predicate S'" exists satisfying: 

Bxo.Hxo) ^ S"'{xo) (10) 

\/x.S"'{x) A G{x) 3x'. A T{x, x') A S"'{x') (11) 

\/x.S"'{x) A -nG{x) -nA{x) (12) 

We are very nearly done: Definition 6 captures that there is some trace con¬ 
taining an error state starting from some initial state. However, our definition of 
an execution trace (Definition 1) includes infinite traces. Thus, the trace contain¬ 
ing the error may be infinite and the error state will not be reachable at all. For 
example, consider Figure 4c. A possible partial danger invariant is ‘true’, which 
meets all of the criteria 10, 11 and 12. However, the program is in fact safe - it 
contains no terminating traces and so the assertion is never even reached. Defi¬ 
nition 6 captures partial danger, which disregards the termination behaviour of 
the program. Instead, what we really want is total danger, which will guarantee 
that there is some finite trace culminating in an error. To ensure that the error 
traces are finite, we will introduce a ranking function, which will serve as a proof 
of termination. Below we recall the definition of a ranking function: 

Definition 7 (Ranking function). A function R : X ^ Y is a ranking func¬ 
tion for the transition relation T ifY is a well-founded set with order > and R 
is injective and monotonically decreasing with respect to T. That is to say: 

\lx,x' € X.T{x,x') R{x) > R{x') 

This is the final piece we need to define danger invariants: 

Definition 8 (Danger Invariant). A pair {D,R) of a predicate and a rank¬ 
ing function is a danger invariant for the loop L{I,G,T, A) iff it satisfies the 
following criteria: 

3xo.I{xo) A D{xo) (13) 

\/x.D{x) A G{x) —s- R{x) > 0 A 3x'.T{x, x) A D{x) A R{x) < R{x) (14) 
Wx.D{x) A -^G{x) —> ^A{x) (15) 

Theorem 1 (Danger Invariants Prove Bugs). The loop L{I,G,T, A) is 
unsafe iff there exists a danger invariant satisfying the criteria in Definition 8. 
In other words, the existence of a danger invariant is a necessary and sufficient 
condition for the reachability of a bug. 

3 Second-Order Formulation of Danger Invariants 

The problem of program verification can be reduced to the problem of finding 
solutions to a second-order constraint [8,9]. In order to give a second-order 
formulation of a danger invariant, we will use a fragment of second-order logic 
decidable over finite domains that we defined in [1], and to whose satisfiability 
problem we refer as Second-Order SAT. Next, we recall this fragment: 
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Definition 9 (Second-Order SAT). 

. . . Sra-Ql^l • • • Qn^n-^ 

WTiere the Si range over predicates, the Qi are either 3 or V, Xi range over 
Boolean values and a is a quantifier-free propositional formula that may refer to 
both the first-order variables Xi and the second-order variables Si. 


Definition 10 (Danger Invariant Formula [DI]). 

3D,R,xo.Sx.3x'.I{xo) A D{xo) A 

D{x) A G{x) T{x, x) A D{x ) A 
R{x) > 0 A R{x) > R{x) A 
D{x) A -^G{x) -^A{x) 

Definition 11 (Skolemized Danger Invariant Formula [SDI]). 

3D,R,N,xo.'ix.I{xo /\ D{xo) A 

D{x) A G{x) R{x) > 0 A T{x, N{x)) A D(N{x)) A R{x) > R{N{x)) 
D{x) A -'G{x) —>■ -^A{x) 


Fig. 5: Existence of a danger invariant as second-order SAT 


Our first second-order formulation of a danger invariant is captured in the 
second-order SAT formula [DI] of Definition 10. In this definition, in order to 
specify that from each D-state we can reach another by iterating the loop once, 
we require quantifier alternation over the first order variables. However, the 
solver from [1] that we want to use requires eliminating the extra level of quan¬ 
tifier alternation (the inner existential quantifier) by using Skolem functions. 

If the transition relation T is deterministic, then we do not need the quantifier 
alternation, since each x has exactly one successor x' . Thus, we can just replace 
the inner 3x' in the formula [DI] by 'ix' . However, if T is non-deterministic, we 
must find a Skolem function N which resolves the non-determinism by telling 
us exactly which successor is to be chosen on each iteration of the loop. This is 
shown in the formula [SDI] of Definition 11. 

3.1 Danger Invariants Generation 

In this section, we discuss how to solve the constraints generated for danger 
invariants. In [1], we show that Second-Order SAT is polynomial-time reducible 
to finite synthesis, and finite-state program synthesis is NEXPTIME-complete. 





8 


Next, we provide a short description of the program synthesis algorithm and, for 
more details, we direct the reader to [1]. Our algorithm is sound and complete 
for the hnite-state synthesis decision problem. In the case that a specification 
is satisfiable, our algorithm produces a minimal satisfying program. We use 
Counterexample Guided Inductive Synthesis (CEGIS) [10,11] to find a program 
satisfying our specification. 


Candidate program 



Counterexample input 


Fig. 6: Abstract synthesis refinement loop 


As illustrated in Figure 6, the algorithm is divided into two procedures: 
SYNTH and VERIF, which interact via a finite set of test vectors inputs. By 
using explicit proof search, symbolic bounded model checking and genetic pro¬ 
gramming with incremental evolution [12,13], the SYNTH procedure tries to find 
an existential witness P that satisfies the partial specification: 

3P.Vx G INPUTS.cr(x, P) 

If SYNTH succeeds in finding a witness P, this witness is a candidate solution 
to the full synthesis formula. We pass this candidate solution to verif which 
determines whether it does satisfy the specification on all inputs by checking 
satisfiability of the verification formula: 

3x.-<a{x, P) 

If this formula is unsatisfiable, the candidate solution is in fact a solution to 
the synthesis formula and so the algorithm terminates. Otherwise, the witness x 
is an input on which the candidate solution fails to meet the specification. This 
witness x is added to the inputs set and the loop iterates again. 


3.2 Generalised Safety-Danger Formula 

The decision procedure introduced in [1] and briefly described above relies on 
a small-model argument to determine that a formula is unsatisfiable and, in 
practice, it is usually unable to prove unsatisfiability. Therefore we would like 
to only provide satisfiable formulae whenever possible. Since the program we 
are analysing is either safe or unsafe, and assuming that a proof is expressible 
in our logic, a program either accepts a safety invariant or a danger invariant. 
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We model this as a disjunction in the formula [GS] of Definition 12. [GS] is a 
theorem of second-order logic, and our decision procedure will always be able 
to find witnesses S', D, TV, S, j/o demonstrating its truth, provided such a witness 
is expressible in our logic. The synthesised predicate S is a purported safety 
invariant and the D, N, R, yo constitute a purported danger invariant. If S is 
really a safety invariant, the program is safe, otherwise D, R (with witnesses to 
the existence of an error trace with Skolem function N and initial state j/o) will 
be a danger invariant and the program is unsafe. Exactly one of these proofs will 
be valid, i.e. either S will satisfy the criteria for a safety invariant, or D, N, R, t/o 
will satisfy the criteria for a danger invariant. We can simply check both cases 
and discard whichever “proof” is incorrect. 


Definition 12 (Generalised Safety Formnla [GS]). 


35, D, N, R, yoS/x, x , y. 


(I{x) —>■ S{x) A \ 

S{x)/\G{x) AT{x,x') ^ S{x') a \ V 
\ S{x) A -'G{x) —>■ A{x) j 

/ I{yo) A D{yo) A \ 

D{y) A G{y) -A R{y) > 0 A T{y, N(y)) A D(N(y)) 
A R{y) > R{N{y))A 

\ D{y) A ^G{y) -A -^A{y) / 


Fig. 7: General second-order SAT formula characterising safety 


4 Illustrative Examples 

To illustrate the use of danger invariants for compactly representing danger 
proofs and finding deep bugs, as well as the shortcomings of the existing bug 
finding techniques, we consider the programs in Figure 8, where (a), (b), (c), (d) 
contain deep bugs, whereas (e) models a buffer overflow. 

For program (a), any execution trace violates the assertion unless the nonde- 
terministic choice (denoted by “*”) is such that y is incremented exactly 999999 
times out of the 1000000 iterations of the loop. In order to analyse this program 
bounded model checkers have to completely unwind the loop. The resulting SAT 
instance is very large, so large in fact that solving it will almost certainly take 
far too long to be practical. Hybrid approaches such as predicate abstraction 
and LAwI will have to progressively unwind the loop in order to refute spurious 
counterexamples of increasing length. Again, this is most likely to take too long 
to be practical. 
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In our case, in order to prove that the assertion can be violated, we need to 
hnd a danger invariant. This means that we need an approximation of the set 
of states reachable during the program’s execution that must include a feasible 
counterexample trace. Of course there may be several such invariants. One pos¬ 
sibility for this example is D{x,y) = x < y and ranking function R{x,y) = —x. 
D holds in the initial state where a; = 0 and y = 1, and it is inductive with re¬ 
spective to the loop’s body if the nondeterministic choice is given by the Skolem 
function Ny(x, y) = y + \. That is: 

Vx, y, x' .X <y^x' = x + l/\x'< Ny{x, y) 

Program (b) is a version of (a) with additional nondeterminism. Now, in each 
loop iteration, x may or may not be incremented. This modification substantially 
increases the number of reachable program states, as well as introducing a po¬ 
tential non-terminating behaviour as x may not reach 1000000. As a result, 
bounded model checkers will loop forever trying to generate the SAT instance 
corresponding to the unwound loop. Similarly, predicate abstraction and LAwI 
based approaches are even less likely to be practical than for version (a) of the 
program due to the increase in the number of reachable states. We synthesise 
the same D as for program (a) as it still contains a feasible counterexample trace 
regardless of the additional nondeterminism. Here, termination depends on the 
Skolem function giving us the successor of x. Thus, we can have R{x, y) = —x 
and Nx{x, y) = a; -I- 1. 

Program (c) is similar to (a), with the exception that the assertion is now 
negated. This example is more intricate as the danger invariant needs to capture 
the evolution of x and y from the the initial state where they are not equal to a 
hnal state where there are (and hence they cause the assertion to fail). One such 
invariant is D{x,y) = y == (x < 1?1 : x) and R{x,y) = —x. Essentially, this 
invariant says that y must not be incremented for the first iteration of the loop 
(until X reaches the value 1), and from that point, for the rest of the iterations, 
y gets always incremented such that x == y. For this case, I? is a compact and 
elegant representation of exactly one feasible counterexample trace. The witness 
Skolem function that we get is Ny(x, y) = {x < l?y : y + 1). 

Program (d) is taken from [14], and it is meant to illustrate the difficulty 
faced by tools based on predicate abstraction and abstraction refinement with 
deterministic loops with a fixed execution count as many iterations of the iter¬ 
ative refinement algorithm correspond to spurious executions of the loop body. 
The assertion fails, but proving this requires 1000000 iterations of the rehne- 
ment loop, resulting in the introduction of the predicates (i == 0),(f == 
l),...,(i == 1000000) one by one. For us, some of the possible danger invari¬ 
ants are D{i,c,a) = a< — 10, D{i,c,a) = a == 0, D{i,c,a) = a<0 and 
R{i, c, a) = true. 

Program (e) models a check for a buffer overflow. The buffer overflow does 
happen whenever x is big enough such that the computation of a; =i= 4 overflows. 
We find the danger invariant D{x,i,len) = {i == 0 A len == 0 A x 0) and 
ranking function R{x, i, len) = true. This basically says that the computation of 
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len overflowed resulting in len = 0 (while x 0). Consequently, the loop is not 
taken such that i stays 0 and the ranking function is true. Note that this example 
has the assertion inside the loop, and required some trivial preprocessing in order 
to move it outside the loop. 


X = 0; y = 1; 

while (x < 1000000) { 

X++; 

if (*) y++; 


assert ( x = y ) ; 

(a) 


X = 0; y = 1; 

while (x < 1000000) { 

if (>1=) X++; 

if (*) y++; 


assert ( x = y ) ; 

(b) 


void foo(int a) 

{ 

int i, c; 
i = 0; 
c = 0; 

while (i < 1000000) { 
c=c+i ; 
i = i +1; 

} 

assert(a > 0); 

} 

(d) Adapted from [14] 

int main(void) { 

unsigned int x, i , len; 


X = 0; y = 1; 

while (x < 1000000) { 

X++; 

if (*) y++; 


len = X * 4; 

for ( i = 0; i < x; i++) { 
assert(i * 4< len) 

} 


assert ( x != y ) 


(c) 


(e) Checking for a buffer overflow caused 
by an integer overflow. 


Fig. 8: Motivational examples. 


5 Danger Invariants in Relation to Other Formalisms 

In this section, we relate danger invariants to other existing formalisms. For 
this purpose, we start by providing a characterisation of danger invariants as a 
fixed point computation. First, we define the set of program executions starting 
in an initial state, Ef^jd, and the set of program executions ending in an error 
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state, Ehck- The set of program execution traces starting in an initial state (real 
executions) is 

'Efwd — — I T'(Xj , ) A Xq € 

where Ff^d constructs execution traces by taking a transition forward: 

Ffwd = \s.{{x) I X e /} U {(xq. • -^n;^n+1 ) I (xo...x„) e5AT(x„,x„+i)} 
The set of program execution traces ending in an error state is 

F'bck — I X'(Xj,Xi+l) /\ Xyi € F^ — ^fpi^F^ck^ 

where Tbcfe constructs execution traces by taking a transition backward: 

Fbck = \S.{{x) I X € £;} U {(x_i,Xo...x„) I (xo...x„) e S' A r(x_i,Xo)} 

Similar to [15], we can now express the set of execution traces starting in an 
initial state and ending in an error state as Frr = lfp{Ffwd) H lfp{Fbck)- Then, 
a danger invariant is an approximation of lfp{Ffj^d) (either D D lfp{Ffwd) or 
D C lfp[Ffyjd)), such that it contains at least one error trace D n Frr ^ 0. 

5.1 Abstract Interpretation 

Given that direct computation of the set of possible execution traces of a program 
is most of the times infeasible, an abstract interpretation based analysis conser¬ 
vatively computes at each program point a set of abstract states representing 
an overapproximation of the possible concrete program states [2]. The analysis 
assigns to each program location an abstract value from an abstract domain. 
The concretisation mapping 7 is defined such that for every program location I 
produces an abstract state x^, such that 7 (x^) contains all the concrete states 
reachable at location 1. 

The abstract forward interpreter Ff^d"^ is inherently overapproximating, 
lfp{Ffyjd) C jilfp^{Ffj^d^)). This means that, whenever an error is signalled at 
program location I, the alarm may be spurious. In order to decide whether an 
alarm is genuine, abstract interpretation based techniques require backward anal¬ 
ysis starting from the error state. Various solutions have been already proposed 
in the abstract interpretation literature [16,17,15]. Next, we see which of these 
can be used to compute danger invariants. 

For illustration purposes, we will use the program in Figure 9 (adapted from 
[16]) as a running example throughout this section. The program takes an input 
variable y bounded between 100 and 200 and iteratively decreases it by 2. The 
program is erroneous as the assertion y == 0 is violated whenever the initial 
value of y is odd. The abstract states xo** to X4* inferred by a forward analysis 
based on the interval domain are listed next: xo* = [ 100 , 200 ], xi* = [— 1 , 200 ], 
X 2 “ = [1, 200], X 3 “ = [-1,198], X4# = [-1, 0]. 

Since X 4 ** violates the assertion y == 0, in order to check whether it is 
genuine, the negation of the assertion y<0Ay>0 must be propagated backwards. 
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Given that the result of the forward analysis is a sound overapproximation, the 
negation of the assertion can be intersected with X 4 ^ resulting in j/ = — 1, which 
is then propagated backwards. If this propagation results in the set of states 
being empty at any program location, then the error is a false alarm. Otherwise, 
it is genuine. The main challenge is the presence of loops. Given a state after a 
loop, it is non-trivial to infer a state that is valid prior to entering the loop. In 
particular, it is necessary to assess how often the loop body needs to be executed 
to reach the exit state. 

In [16] Brauer and Simon use an overapproximating affine analysis to estimate 
the number of loop iterations. Thus, they are able to obtain y = 2n—lAn S [1,63] 
at location li (corresponding to xi^ in the CFG). In order to obtain a danger 
invariant from this, we just need to add the error state y = —1. Thus, we 
obtain D{y) = {y == 2n — 1 A n € [0, 63]), which is is guaranteed to contain a 
concrete counterexample. In [17], Erez performs a bounded search for backward 
traces up to a given depth. For the given example, the first counterexample 
found is: {y = —l,y = 1, ■■■,y = 101). Thus, a corresponding danger invariant is 
D{y) = {y == —iVy == lV...Vj/ == 101), or D{y) = {y == 2n—lAn G [0, 51]). 
While the results of both these techniques can be immediately used to compute 
danger invariants, this is not true for other works on eliminating false alarms in 
abstract interpretation based techniques such as [15], where Rival computes an 
overapproximation of Err by using in his backward analysis the same domains 
as in forward analysis. Thus, the result is not a danger invariant and cannot 
ensure the existence of a true error. 



Fig. 9: The CFG of an erroneous program: the labels on the vertices denote the 
corresponding abstract program states and the arcs correspond to instructions 

in the program. 


5.2 Bounded Model Checking 

For a loop L{I,G,T, A), a bounded model checker progressively unwinds the 
transition relation up to a depth k. As such: 
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where constructs execution traces by taking a transition forward: 

Ffind = I a; €/} U {(xo...x„,x„+i) | (xo—Xn) € S AT{x„,Xn+i) /\n<k} 

Thus, BMC finds counterexample traces of the form {xq, where xq € 

I, Xn & E and n<k. Such a counterexample directly corresponds to the danger 
invariant D{x) = Vi=o n ^he running example, we have used CBMC [18] 

and obtained the counterexample trace {y = 101,?/ = 99,...,y = —1), which 
corresponds to the danger invariant D{y) = {y == 101 V y == 99 V ... V y == 
-!)• 


5.3 Linear Invariants 

There is a lot of work on the generation of linear invariants of the form ciXi + 
... + Cndn + d < 0 [19, 20]. The main idea behind these techniques is to treat 
the coefficients ci,..., c„, d as unknowns and generate constraints on them such 
that any solution corresponds to a safety invariant. In [20], Colon et al. present 
a method based on Farkas’ Lemma, which synthesises linear invariants by ex¬ 
tracting non-linear constraints on the coefficients of a target invariant from a 
program. In a different work, Sharma and Aiken use randomised search to find 
the coefficients [20]. It would be interesting to investigate how these methods 
can be adapted for generating constraints on the coefficients ci,... ,c„,d such 
that solutions correspond to linear danger invariants. 

6 Experimental Results 

To evaluate our algorithm, we implemented the Dangerzone tool, which gen¬ 
erates a danger specification from a C program and calls the second-order SAT 
solver discussed in [1] to obtain a proof. We ran the resulting prover on 20 buggy 
programs including the running examples in the paper, some examples from the 
literature and some from SV-COMP’15 [21]. Our benchmarks do not make use 
of arrays or recursion. We do not have arrays in our logic and we had not im¬ 
plemented recursion in our frontend (although the latter can be syntactically 
rewritten to our input format). 

For each benchmark we infer a danger invariant, a ranking function, an initial 
state and Skolem functions witnessing the nondeterminism. To provide a com¬ 
parison point, we also ran CBMC [18] on the same benchmarks. For CBMC, we 
manually provided sufficient unwinding limits for each program. Each tool was 
given a time limit of 1800 s, and was run on an unloaded 4-core 3.30 GHz i5-2500k 
with 8 GB of RAM. The results of these experiments are given in Figure 10. 

On programs with shallow bugs (i.e. bugs that can be reached after a small 
number of loop iterations), CBMC is much faster than Dangerzone. How¬ 
ever, Dangerzone performs much better on programs with deep bugs provid¬ 
ing empirical evidence that danger invariants improve the scalability of static 
bug finding. We feel that the performance difference for shallow bugs is inherent 
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to the difference in the two approaches - our solver is more general (and less 
engineered) than CBMC, in that it provides a complete proof system for both 
danger and safety. The two examples on which Dangerzone times out have an 
intricate control flow structure which requires specific constants to be inferred in 
the danger invariant - a case for which Dangerzone is not optimised. Notably, 
only 6 of the benchmarks (marked with * in the table) contain doomed loop 
heads. 


Benchmark 

Deep Bugs 

Shallow Bugs 

CBMC 

Dangerzone 

loopl.c* 

/ 

- 

575 s 

4.32 s 

loop2.c 

/ 

- 

TO 

6.64 s 

loops.c [Fig 8c] 

/ 

- 

TO 

76.58 s 

loopd.c [Fig 8e] 

- 

/ 

0.082 s 

8.82 s 

loop5.c [Fig 8d] 

/ 

- 

TO 

8.89 s 

loop6.c [14] 

- 

/ 

0.084 s 

6.51s 

loop7.c* 

/ 

- 

TO 

8.42 s 

loops.c* 

/ 

- 

TO 

5.09 s 

loopQ.c* 

/ 

- 

TO 

9.68 s 

looplO.c 

/ 

- 

TO 

6.59 s 

loopll.c [Fig 9] 

- 

/ 

0.171s 

5.54 s 

loopl2.c* 

- 

/ 

0.081 s 

12.09 s 

looplS.c 

- 

/ 

0.256 s 

7.22 s 

loopld.c 

- 

/ 

0.286 s 

TO 

loopl5.c 

- 

/ 

2.111s 

TO 

looplG.c* 

- 

/ 

0.081 s 

8.20 s 

loopl7.c [Fig 8a] 

/ 

- 

TO 

6.30 s 

looplS.c 

/ 

- 

TO 

8.59 s 

loopl9.c [Fig 8b] 

/ 

- 

TO 

7.40 s 

loop20.c [Fig 4a] 

- 

/ 

0.082 s 

6.97 s 


Key: TO = time-out 


Fig. 10: Experimental results 


7 Conclusions 

In this paper, we introduced the concept of danger invariants - the dual to safety 
invariants. Danger invariants summarise sets of traces that are guaranteed to 
reach an error state. This summarisation allows us to find deep bugs without 
false alarms and without explicitly unwinding loops. This new concept promises 
to deliver the performance of abstract interpretation together with the concrete 
precision of BMC. We presented a second-order formulation of danger invariants 
and used the solver described in [1] to compute danger invariants for a set of 
benchmarks. 
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